From a packaging perspective, this just shifts the problem. Just like hadrian, cabal-install's dependency closure is not negligible and it tends to be picky about dependency versions (sometimes even unnecessarily so). Bootstrapping GHC with Hadrian is bad since it involves a nontrivial amount of work getting its dependencies compiled. With this solution, the problem would just be shifted to bootstrapping cabal-install.

Cabal the library has the advantage that it is necessarily available since it's bundled (for better or worse) with your bootstrap GHC which you need anyways. As a consequence, I'd suggest

• Either writing a small wrapper around Cabal which implements the functionality necessary for this build system (which is probably preferrable to outright linking against it since that would make it harder to switch to a Makefile later).
• Use runghc Setup.hs instead of cabal-install. It looks like this would not work with the current code since you rely on e.g. project files which aren't supported. Also it carries some risk since Setup.hs isn't treated with the same amount of care as cabal-install by the upstream maintainers (so bigger refactors carry the risk of regressing Setup.hs which isn't always caught before a release).

@sternenseemann I would simply do haskell.nix or cabal2nkx on it, vender the generated nix files, and cut out cabal-install that way.

@sternenseemann this is actually a good point hmm, dunno why it hadn't rung any alarm for me earlier.

Well I am less worried about building cabal-install itself, though okay there is that too (I don't really think the deps are significantly worse than hadrian, and the big plus is it is decoupled from ghc unlike hadrian).
My concern now is rather that cabal-install typically needs a network connection at runtime.
But maybe it is possible to run it or make it run off-line too (Fedora does that for rust cargo)??
Otherwise distros would actually be in a worse position than with hadrian, since they need Cabal not cabal-install, so I agree with you there.

My concern now is rather that cabal-install typically needs a network connection at runtime.

This is effectively disabled by the setting active-repositories: :none in the generated cabal.project file which means that cabal would only use the pre existing package db. You can also pass --offline, but in my experience cabal frequently ignores this flag.

Well I am less worried about building cabal-install itself, though okay there is that too (I don't really think the deps are significantly worse than hadrian, and the big plus is it is decoupled from ghc unlike hadrian).

I suppose this hurts nixpkgs more than others. We always need to fully bootstrap anything, so we can never rely on a preexisting cabal-install that was built by our previous GHC package. (I'm not sure if Fedora does this, but I do know that Alpine heavily relies on this, also for the stage 0 GHC.) As a result, we need to build everything needed for bootstrapping GHC using the upstream bindists (or whatever bootstrap compiler we use). This can be finnicky and unreliable, so it is preferrable if less packages have to be built. Also, we try to use the GHC core packages as much as possible (like Stackage) which makes building cabal-install a nightmare because it has become really picky about filepath and directory as of late. Having support for a wide range of core package versions no longer seems to be a concern upstream (in fact, I've given up getting cabal to work for all GHC versions.)

I do agree, though, that cabal-install is not as bad as Hadrian. It's dependency closure is probably similar, but we would not need to compile cabal-install with stage0, specifically.

I would simply do haskell.nix or cabal2nkx on it, vender the generated nix files, and cut out cabal-install that way.

Well, we use Setup.hs for “cutting out cabal-install“ which is, of course, troublesome when it comes to stuff like project files which it doesn't support. I'll have to read through Build.hs and figure out whether cabal-install is used to manage the package-db for compiling and what the project file is used for, I guess.

@sternenseemann they way I see it, Build.hs is pure tech debt, and intentionally so :) The problem with Hadrian is that it makes it "too easy and nice" to write custom logic for building GHC, whereas here the goal is to make sure GHC is "just" a bunch of cabal packages, and anything beyond that is awkard and sticks out like a sore thumb.

cabal and cabal-install are huge code bases, but I hope the relevant declarative parts of the cabal files here are much simpler and amendable to more implementation strategies --- i.e. it would be also bad if this ended up using a huge long tail of random cabal-install features, but I don't think that will be the case.

CC @andreabedini who is always talking about how "we shouldn't need bespoke Haskell tools to build Haskell projects".

I've been tasked with building a GHC bindist with cabal-install instead of Hadrian but I don't have more insights on what is planned for distributions like Fedora or Nixpkgs after that. At least we would be relying on a properly released tool, which Hadrian isn't, so it should be a step in the right direction. It is definitely a step in the direction of reinstallable boot libraries to make the compiler multi-target, but that's another story.

Project files are used to pass options to build packages (cabal flags, CPP defines...) and to manage dependencies automatically. For now we only download happy and alex but we might as well vendor them in the future.

GHC is not fully bootstrappable, we need to start with some binary blob anyway. As such I am not at all concerned about bootstrapping with a binary blob of cabal-install either. And indeed, the bindists we will build will contain a cabal-install executable. There will be static ones. A long term plan may be to use cabal, and derive a ninja, or nix solution alongside, as part of further cabal development. You'd still need some ghc bootstrap blob though.

Hadrian is a failure in so many respects. The biggest for me is that the haskell project, namely its compiler, effectively states, that cabal is not good enough to build it. And as such we spend countless hours and resources on a haphazardly build "alternative", instead of collectively improving cabal for everyone in the community. It's such a waste of resources and NIH syndrome. Another is that Hadrian follows closely the insanity we had with the Make base build system. GHC itself is not properly isolated, there is lots of reaching around packages, instead of properly packaging stuff. @Ericson2314 has addresses one of the long standing global configure issues. There are more similar issues where GHC is effectively treated as one massive amalgamated codebase, instead of a tree of dependencies.

Ultimately the whole symbiotic development of GHC and its build tool should be broken by separating both clearly, on top of improving cabal, which dearly needs a lot of improvements. These improvements should then help everyone on the community, instead of being features only found in Hadrian.

Hadrian must die, or at best be relegated to a tool for GHC developers only; it has served its purpose as a spring board from the old Make base build system to a more cabal'ified version. The logical conclusion is to build GHC with cabal, and improve cabal to the point where this is a no-brainer.

Build.hs is a temporary crutch, ultimately this should be cabal-install, as well as short Makefile, and maybe some tiny shell scripts to glue things together where patching/changing cabal-install is not sensible.

And lastly, just to make sure this is not misunderstood, all changes to GHC and cabal we make, we intent to upstream. This is not supposed to be a fork!